The present invention relates to digital video signal processing, and more particularly to architectures and methods for digital camera image pipeline.
Imaging and video capabilities have become the trend in consumer electronics. Digital cameras, digital camcorders, and video cellular phones are common, and many other new gadgets are evolving in the market. Advances in large resolution CCD/CMOS sensors coupled with the availability of low-power digital signal processors (DSPs) has led to the development of digital cameras with both high resolution image and short audio/visual clip capabilities. The high resolution (e.g., sensor with a 2560×1920 pixel array) provides quality offered by traditional film cameras.
FIG. 2a is a typical functional block diagram for digital camera control and image processing (the “image pipeline”). The automatic focus, automatic exposure, and automatic white balancing are referred to as the 3A functions; and the image processing includes functions such as color filter array (CFA) interpolation, gamma correction, white balancing, color space correction and conversion, and JPEG/MPEG compression/decompression (JPEG for single images and MPEG for video clips). Typical digital cameras provide a capture mode with full resolution image or audio/visual clip processing plus compression and storage, a preview mode with lower resolution processing for immediate display, and a playback mode for displaying stored images or audio/visual clips.
The usual camera has a color CCD or CMOS sensor which includes a rectangular array of photosites (pixels) with each photosite covered by a filter (the color filter array, or CFA). The filter colors may be red, green, or blue (RGB). In the commonly-used Bayer pattern CFA one-half of the photosites are green, one-quarter are red, and one-quarter are blue.
However, the RGB color values obtained from a camera sensor are device-dependent. Thus there is a need to characterize these values in a device-independent fashion and provide a color correction (RGB-to-RGB color correction). For simplicity, common methods presume a linear transformation to perform the color correction:
      [                                        R            ′                                                            G            ′                                                            B            ′                                ]    =                    [                                                            M                11                                                                    M                12                                                                    M                13                                                                                        M                21                                                                    M                22                                                                    M                23                                                                                        M                31                                                                    M                32                                                                    M                33                                                    ]            ⁡              [                                            R                                                          G                                                          B                                      ]              +          [                                                  O              1                                                                          O              2                                                                          O              3                                          ]      
The problem translates to finding the transformation matrix (Mi,j) and the offset vector (Oj). Note that the diagonal matrix elements, Mkk, are typically in the range from 0.5 to 2.0, and that the off-diagonal elements are typically in the range −0.5 to +0.5.
One known approach performs a least-squares optimization without any constraint; see Vrhel, M. J. and Trussell, H. J., Optimal Scanning Filters Using Spectral Reflectance Information, Proc. SPIE Int. Soc. Opt. Eng. 1913, pp. 404-412 (1993). This approach treats all colors equally, and the resulting transform usually loses white balance. However, human eyes are very sensitive to white balance in a color image.
Another known approach uses a white-preserving constraint in the optimization; see Finlayson, G. D. and Drew, M. S., Constrained Least-Squares Regression in Color Spaces, Journal of Electronic Imaging 6(4), pp. 484-493 (1997). This approach keeps white balance and the white point remains unchanged after the transform. This requires that source data and reference data have the same exposure values. However, source data and reference data usually have different exposure values, and exposure information is either unavailable or inaccurate. An exposure value difference leads to larger color errors.
Thus there is a problem with known color correction methods, and with camera tuning tools which incorporate color correction/conversion.